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Abstract 

This paper describes a malaria knowledge test (MKT) developed to evaluate a web-based game for 
students who increasingly travel to malaria-risk regions of the world. The 18-item MKT was 
structured according to the dimensions of the self -regulation model (SRM) to measure the 
accuracy of students' beliefs about malaria. An experimental design was used to compare three 
game versions. Students (N=482) participated in 2010 by completing a pre-test, playing a Web- 
based game simulating student travel to malaria-endemic destinations, and completing a post-test. 
Study data support the validity and reliability of the MKT for the evaluation of malaria education 
interventions and for student self-assessment. Use of the MKT to evaluate an educational game 
about malaria revealed a strong overall learning effect and discrimination by game version, travel 
experience, and SRM dimension. This 5-min test may also be adapted for educational outreach 
purposes among health care providers globally, residents of malaria-endemic regions, and other 
high risk travel groups {e.g., elderly, chronic health conditions, pregnant, or returning to malaria- 
endemic regions to visit friends/relatives). 
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Introduction 

As student travel to malaria-risk regions increases, 1 malaria education becomes a more 
pressing need because infection can rapidly lead to permanent disability or death. 2 Many 
travelers have limited knowledge of malaria despite it being a leading cause of death 
worldwide, with 200-300 million infections and nearly one million deaths each year. 2,3 In 
the absence of a malaria vaccine, risk-reduction depends upon the consistent use of personal 
protection measures and immediate response to the development of symptoms. 4 Students 
who become ill while abroad disrupt activities and unnecessarily burden local health care 
systems. When symptoms develop after returning home, clinicians may be less familiar with 
the complexities of malaria diagnosis and treatment, and malaria parasites may be 
introduced into areas previously cleared of transmission. 3 Factors favoring malaria 
transmission include a low knowledge of malaria and the discomfort, inconvenience, and 
cost of personal protection measures. 



©Copyright L.B. Hartjes and L.C. Baumann, 2012 

Correspondence: Laurie Hartjes, 530 Linden Court Verona, WI 53593, USA. Tel. +1.608.772.9656 - Fax: +1.608.263.5458. 
lhartjes@wisc.edu. 

Contributions: LH, designed the study and was responsible for iterative and summative data collection and analysis; LB, provided 
ongoing consultation during the development and evaluation of the malaria knowledge test (MKT) and the educational intervention 
(malaria game). Both authors participated in writing the manuscript and have approved the final submitted version. 

Conflict of interests: the authors declare no conflict of interests. 



Hartjes and Baumann 



Page 2 



This paper describes a malaria knowledge test (MKT) developed to test the efficacy of an 
educational game (a web-based travel-simulation that informs students about malaria 
through scenario-based decision-making and outcomes that include health, illness, disability 
or death). 5,6 MKT development was undertaken when a suitable knowledge test was not 
identified in the literature. Youthful travelers are uniquely vulnerable because they have less 
experience with the atypical characteristics of many tropical diseases, ' 7 and they are 
susceptible to the developmentally normative risk-taking behavior characteristic of 
adolescents and young adults. 12,13 

Materials and Methods 

Although a more detailed description of the overall experimental design, participant sample 
and MKT items is provided elsewhere, 5,6 the highlights of MKT development and 
evaluation are reviewed in this report. Following iterative testing of the MKT and an 
associated game, a summative evaluation was conducted using an experimental design that 
compared three game versions that varied player access to feedback in response to game 
decisions. Feedback was chosen as the factor of interest because there is limited evidence 
about which feedback strategies are most effective in the context of brief-exposure 
educational games, particularly those addressing complex topics with potentially serious 
consequences if content is misinterpreted. Each treatment group was exposed to one of three 
types of explanatory feedback: i) a rationale for why a game action had a positive or 
negative effect, provided automatically as a pop-up text box after each player action (in- 
game decision); ii) a rationale accessed at player-discretion by clicking on a link associated 
with that action; and iii) no access to a pop-up rationale. Students were recruited from seven 
Midwestern U.S. campuses. Study abroad program administrators distributed information 
about the study using internal e-mail lists; study co-investigators had no direct access to 
these email lists. 

Procedure 

Approval was obtained from the Institutional Review Board on each campus prior to study 
deployment. A link within the invitation letter was used to access the study protocol after 
participants read the informed consent information. Consent was implied when students 
clicked on the study link, and participation could be terminated at any time. Participants 
were linked sequentially from the pre-game survey to the game and then to the post-game 
survey. Following post-game survey submission, participants were directed to a summary 
site where they were provided with their pre/post MKT scores and additional malaria 
resources. 

Development of the malaria knowledge test 

The MKT was structured using the dimensions of the self-regulation model (SRM) 5,14 and 
content consistent with US and international standards for malaria education. 2 " 4,7 The SRM 
is centered on the personal construction of meaning that is processed both cognitively and 
emotionally. 14 The SRM proposes that emotions influence behaviors differently than 
cognitive representations, and are a crucial component of theories to predict how individuals 
respond to perceived health risks. 15,16 Cognitive representations are assessed within the 
following dimensions in response to a perceived health threat: identity (symptoms and 
labels), cause, timeline, consequences (seriousness), control and/or cure, and coherence 
(overall understanding). Emotional representations are examined through the measurement 
of an individual's degree of fear, anxiety, or worry related to a health threat. 

Individual responses to health challenges are determined by personal beliefs and 
expectations. 14 Coping activities are undertaken in response to health beliefs and the worry 
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or anxiety elicited by these beliefs. 6 In applying the SRM to malaria, an accurate 
understanding of malaria risk and prevention options is needed to modulate an appropriate 
emotional response to malaria as a health threat. This emotional processing may then prompt 
risk reduction actions, such as a rapid response to potential malaria symptoms. It is 
important to improve representational accuracy for a complex disease such as malaria that 
has serious consequences and is prevalent in many areas of the world, yet is largely 
unfamiliar to individuals from non-endemic countries. 

The process of enhancing the accuracy of personal representations takes time and often 
repeated exposures because health beliefs can be resistant to change. 14,15 According to the 
SRM, a student who believes that malaria is no big deal (i.e., unlikely and not worrisome) 
will have minimal motivation to engage in prevention behaviors. When this student 
experiences debilitating symptoms secondary to blood stage malaria infection, or learns of a 
tragic malaria outcome for a peer, this new information impacts the student's representation 
for malaria and may increase motivation to engage in prevention behaviors. Conversely, 
returning from travel to a malaria-endemic country with no apparent symptoms can 
reinforce inaccurate information about malaria, such as transmission being rare and/or not 
serious, and may reduce motivation to seek testing for potential malaria symptoms during 
the year following risk exposure. Learning primarily from personal experience may be 
dangerous in the case of malaria. However simulated experience offers a safe approach for 
the refinement of cognitive and emotional representations through trial and error, with 
feedback provided that is specific to the actions taken. 

The MKT was constructed to gauge the accuracy of malaria knowledge in a study abroad 
student population. Further, there is evidence that learning experiences that increase an 
individual's emotional engagement and perception of risk are more likely to change beliefs 
and behavior than abstract information. 17 ' 18 To measure the emotional representation for 
malaria, students were asked to rate their degree of worry about being infected during travel 
to a malaria-endemic region. This item was part of a risk perception assessment that also 
included perceived likelihood of being infected with malaria. 

A pilot test of the initial 12-item MKT was administered on paper with 13 students who 
were close to departure for a field experience in a malaria-endemic location. The mean age 
was 22 years, with about half (54%) reporting graduate student status and all recently 
receiving classroom education about malaria prevention. The MKT items were scored 0 for 
incorrect and 1 for correct; the overall mean score was 91% correct. Revisions to increase 
consistency and flow included adding six questions to achieve three per SRM dimension and 
consistently offering five response options in a multiple choice format (three true/false 
questions were restructured). 

A second pilot test was conducted over the Internet using Zoomerang survey software 
(MarketTools, San Francisco, CA, USA) with 36 pre-nursing undergraduate students who 
had no prior classroom exposure to malaria education. Participants were primarily female 
(97%) and had a mean age of 19 years (range 18-21). Eighty percent planned to study 
abroad, 11% had already returned from a study abroad experience, and 73% had previously 
traveled outside the United States unrelated to a study abroad experience. The mean MKT 
score was 50%. Following this second pilot test, three cognitive interviews were conducted 
with students to further explore their interpretation of the meaning of question stems and 
response options, and what they liked most and least about content and formatting. 
Following minor text revisions, face validity was tested with five study abroad student 
reviewers. Content validity was tested for items within each dimension of the SRM with five 
travel health specialists, and a survey research specialist suggested formatting edits to 
enhance online comprehension. 
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The final pre-game survey consisted of an 18-item multiple-choice MKT with five response 
options (Appendix A), and two items that assessed perceptions about the likelihood and 
degree of worry related to malaria infection that could be acquired during travel to a 
malaria-endemic region (Appendix B). The post-game survey additionally included nine 
demographic items, one player satisfaction item, and an open-ended text box for comments. 

Results 

First, select findings from the comparison of game versions are provided because the MKT 
capacity to detect knowledge differences between groups (by pre/post knowledge gain, 
game version, past travel, SRM dimension) is central to the evaluation of MKT efficacy. 
Second, an SRM dimensional analysis and examination of responses to individual MKT 
items are reported in order to inform future malaria risk reduction interventions. 

Demographic data is summarized in Table 1 for the 482 participants who completed the 
study protocol. MKT mean completion time was approximately 5 min. The Kuder- 
Richardson 20 (KR-20) coefficient of reliability for measures with dichotomous choices 
(correct/incorrect) was 0.82 pre-game and 0.73 post-game. The pre-game MKT mean score 
was 6.6 (37% correct, SD=4.2) and the post-game mean score was 13.8 (77% correct, 
SD=2.8). The pre/post comparison (f=39.38, df=481, P<0.001) and large effect size (0.767) 
demonstrated a strong overall learning effect. 

Pre/post difference scores on the MKT were also compared by game version. The group that 
automatically received explanatory feedback after each game action showed the greatest 
knowledge gain (F(2482)=6.5, P<0.005). There were no differences in knowledge gain by 
study abroad status or by frequency of playing computer games. Perceptions of worry about 
malaria infection increased from pre- to post-game (f=3.54, df=477, P<0.001, 2-tailed) and 
there were differences for the variables of pre/post knowledge scores and worry ratings 
when examined by travel experience. Participants who responded yes to an item about past 
travel to a malaria-risk region had higher mean knowledge scores both pre-game 
F(2478)=39.85, P<0.0001) and post-game (F(2478)=7.54, P<0.001) compared to those who 
responded no or don 't know, although the malaria-experienced group still answered only 
49% of the pre-game knowledge questions correctly. Participants reporting past travel to a 
malaria-risk region had lower worry ratings pre-game F(2478)=17.90, P<0.01 and post- 
game F(2478)=21.44, P<0.01 as compared to those without this reported risk exposure. 
However worry ratings increased from pre- to post- for both the malaria-experienced 
(f=3.098, df=154, P<0.01) and the malaria-naive (f=2.483, df=298, P<0.01) groups. 

A self -regulation model dimensional analysis revealed that knowledge increased in every 
dimension, although the timeline and consequences dimensions had the lowest mean scores 
at both time -points (Table 2). The pre-test items were examined to determine common errors 
in baseline malaria knowledge. For timeline, 92% were unable to identify the shortest time 
period from infection to symptoms and 67% did not recognize the possibility of a delayed 
symptom onset. For consequences, 86% were unaware that malaria infection may result in 
severe anemia, kidney failure, and brain effects, and 80% were unaware that malaria might 
be rapidly fatal. Additionally, 68% did not know that the infectious agent is a parasite 
{cause) and 82% failed to recognize that the peak biting time for malaria transmission is 
dusk to dawn (coherence). When participants were asked to choose the best prevention 
measure from a list of four actions and one don't know response (control), 49% opted for a 
non-existent vaccine and 25% selected don't know. In contrast, identity was the dimension 
with the greatest gain in representational accuracy after participants completed game play; 
nearly all correctly identified malaria symptoms (97%), the type of diagnostic test (99%), 
and the degree of risk globally (92%). 
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Among respondents reporting past travel to a malaria-risk region on the pre-test, 47% opted 
for a non-existent vaccine and 1 1 % selected don 't know when asked to choose the best 
prevention measure. The post-test responses for this group were examined as a best-case 
scenario for malaria comprehension since these students had experienced both actual and 
simulated exposure to malaria transmission. The lowest MKT scores persisted within the 
SRM dimensions of timeline and consequences. The three items with the most incorrect 
responses on the post-test were: malaria can be rapidly fatal (63%); malaria infection may 
result in severe anemia, kidney failure, and brain effects (46%); symptoms may first be 
noticed months after being infected (44%). 

Discussion 

This section discusses what the game evaluation data reveal about the MKT as an 
assessment instrument, and how the MKT is linked to the larger objective of reducing risk 
among travelers to malaria endemic destinations. The MKT was constructed along the 
dimensions of the SRM to measure the accuracy of malaria beliefs in a study abroad student 
population. Study findings demonstrated acceptable internal consistency (pre and post 
KR-20) and pre/post sensitivity to change for knowledge and worry about malaria. The 
MKT items also successfully discriminated between groups by the game version, by travel 
experience, and by SRM dimension. Pre-game MKT scores indicated that malaria 
knowledge deficits exist, while post-game MKT scores improved across all SRM 
dimensions in both the malaria-experienced and malaria-naive groups. 

This substantial pre/post learning effect is a key precursor for competent adherence to travel 
recommendations. Findings also revealed an increase in the perception of worry about 
malaria after playing the game. The potential for increased worry to motivate risk-reduction 
actions is supported by prior data indicating that risk examined as a feeling was a better 
predictor of risk-reduction behavior than a purely cognitive probability judgment. 17,18 
Participants showed evidence of persistent health belief inaccuracies in the two 
representational dimensions with the lowest pre/post knowledge scores: timeline and 
consequences . These dimensions may be more complex to comprehend, such as the 
potential for a delayed symptom onset or for a rapid progression to permanent disability or 
death once symptoms develop. These illness characteristics of malaria deviate from illnesses 
most familiar to this study population. 

The significant pre/post increase in MKT scores among participants reporting past travel to a 
malaria-endemic region indicates that even malaria-experienced travelers can benefit from 
this type of intervention. The finding that participants with past travel to a risk region had 
significantly higher pre-and post-MKT scores as compared to those without this travel 
experience supports the SRM precept that concrete experiences impact mental 
representations. However, the group with past travel to a risk region still answered only 49% 
of the pre-game knowledge questions correctly. This is consistent with data that has 
identified knowledge gaps in travelers irrespective of travel history. 9,19 " 21 

The pre/post scores were lowest in the SRM dimensions of timeline and consequences for all 
participants irrespective of travel history. Most surprising on the pre-test was the enduring 
belief among nearly half of the malaria-experienced participants that the best malaria 
prevention measure is a non-existent vaccine. Because malaria is a communicable disease 
with non-specific symptoms that can progress rapidly into a life-threatening illness, 
knowledge assessment and education about risk-reduction is of the utmost importance. 

A malaria knowledge test for travelers was not identified in the literature. However a 34- 
item survey was conducted with Taiwanese physicians and nurses in 2008 on the broader 
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topic of mosquito-transmitted disease (malaria, yellow fever, and dengue). 2 The mean 
percentage correct for the six items about malaria was 67%. The malaria item with the most 
erroneous responses (83.2%) addressed the potential for a delayed symptom onset, a 
timeline dimension similar to the findings of this study. The complicated nature of parasite 
physiology in the human body and varying risk by geographic region present challenges to 
health care providers who must provide pre-travel assessment and education; individualized 
risk reduction plans that include vaccines, medication, and chronic care/emergency 
management; and diagnosis and treatment across a spectrum of acuity. The authors of this 
Taiwanese study conclude that clinician knowledge deficits might be remedied through 
increased exposure to international standards for the delivery of travel health services. 

The present study demonstrates how unique features of a health threat may impact the 
measurement of the accuracy of knowledge. Malaria has a complex timeline dimension 
(acute, cyclic, relapsing), and consequences that can range from complete resolution with 
treatment to permanent disability or death if not recognized and treated promptly. Few 
health conditions have similar features, and even fewer have the global prevalence and 
historic significance of malaria. Identifying the malaria content with the greatest likelihood 
for being misunderstood is important when designing educational approaches. Further, the 
comparison of game versions showed that automatically providing explanatory feedback in a 
brief-exposure educational game supports learning better than having learners seek feedback 
as desired or not providing this feedback. These findings provide valuable information for 
health educators and clinicians about the malaria content most likely to be misunderstood or 
under-appreciated, and how to use feedback most effectively in games and simulations to 
increase the accuracy of representations for malaria. 

Study limitations include a recruitment pool originating from one region of the United States 
and a low participation rate (34% of those who accessed the 30-min online protocol 
completed it without incentive). Participants who were female (73%) and white (86%) were 
overrepresented, although this is consistent with national demographic data for study abroad 
students who over the past decade have been on average 65% female and 82% white. 23 This 
study did not test the degree of knowledge retention beyond one post-test, or the impact of 
increased knowledge and worry on subsequent travel-related behavior. 

Conclusions 

The growing population of students who travel to malaria-endemic regions would benefit 
from more accessible and engaging approaches to travel health education, and from a test to 
evaluate malaria knowledge. One such test, the MKT, required about 5 min to complete 
when administered online to students participating in study abroad programs on seven 
Midwestern U.S. campuses. The MKT demonstrated acceptable internal consistency and 
sensitivity to change in response to an educational intervention. Pre-game MKT scores 
revealed malaria knowledge deficits among study abroad students. Game exposure produced 
significant improvements in MKT scores across all dimensions of the SRM in both the 
malaria-experienced and malaria-naive groups. 

Greater understanding of traveler cognitive and emotional representations about malaria will 
enhance the creation of more tailored interventions. The SRM representational dimensions 
most likely to yield inaccurate responses were timeline and consequences . This presents a 
hazard when students do not understand, for example, how quickly malaria can progress to 
death or disability, how parasite drug-resistance or counterfeit drugs can increase risk for a 
tragic outcome even with a rapid diagnosis, or why symptoms might develop after returning 
home. This open-access knowledge test may also be adapted to evaluate other malaria 
education interventions, and for educational outreach among health care providers globally, 
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residents of malaria-endemic regions, and other high risk travel groups (i.e., elderly, chronic 
health conditions, pregnant, or returning to a malaria-endemic region to visit friends/ 
relatives). 
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Participants' characteristics. 



Table 1 



Mean age in years=21 (SD=4.6) 


* 

n 


% 


Gender: female 


352 


73 


Student status 






Undergraduate 


42 1 


92 


Graduate 


35 


7 


Racial/ethnic background 






Asian 


30 


7 


Black or African American 


12 


3 


Hispanic or Latino/a 


10 


2 


Native American or Alaskan Native 


10 


2 


Native Hawaiian or Pacific Islander 


2 


0.4 


White or Caucasian 


393 


86 


Study abroad status 






Interested, but no specific plans 


36 


8 


Planning 


152 


34 


Currently abroad 


76 


17 


Returned 


187 


35 


Prior travel to a malaria-risk region 






Yes 


155 


32 


No 


299 


63 


Don't know 


24 


5 


Future travel to a malaria-risk region 






Yes 


309 


64 


No 


129 


27 


Don't know 


40 


8 


Computer/video game play frequency 






Never or rarely 


239 


51 


At least once a month 


111 


23 


At least once a week 


62 


13 


Several times a week 


40 


8 


Daily 


24 


5 



n, may vary due to missing cases. 
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Table 2 

Malaria knowledge test scores by self-regulation model (SRM) dimension (N=482, 3 items per SRM 
dimension). 



Dimension 


Pre-test mean (SD) 


Post-test 


Identity 


1.5 (1.13) 


2.9 (0.43) 


Cause 


1.4 (0.87) 


2.5 (0.64) 


Timeline 


0.5 (0.76) 


1.8 (1.02) 


Consequences 


0.8 (0.88) 


1.7 (0.85) 


Control 


1.3(0.95) 


2.6 (0.66) 


Coherence of understanding 


1.0(1.00) 


2.3 (0.82) 
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